Asynchronous video interview system

ABSTRACT

Aspects of an asynchronous video interview system and related techniques include a server that receives a plurality of pre-recorded video prompts, generates an interview script, transmits a video prompt from the interview script to be displayed at a client computing device, and receives a streamed video response from the client computing device. The server can perform algorithmic analysis on content of the video response. In another aspect, a server obtains response preference data indicating a timing parameter for a response. In another aspect, a video prompt and an information supplement (e.g., a news item) that relates to the content of the video prompt are transmitted. In another aspect, a server automatically selects a video prompt (e.g., a follow-up question) to be displayed at the client computing device (e.g., based on a response or information about an interviewee).

CROSS-REFERENCE TO RELATED APPLICATION

This application claims the benefit of U.S. Provisional PatentApplication No. 61/602,566, filed Feb. 23, 2012, the entire disclosureof which is incorporated herein by reference.

BACKGROUND

In spite of the many recent advances in information technology, live,in-person interviews are still required for job applicants, universityapplicants, and the like. In-person interviews suffer from manydrawbacks. They require scheduling (and often rescheduling) that musttake into account office hours, travel time, and time zones, with aconstant risk of cancellations and delays. They also are prone tounintended variation (e.g., by asking the same questions in differentways that elicit different responses) and bias (e.g., disfavoringcandidates interviewed when the interviewer is tired). Further,in-person interviews tend not to be recorded or remembered accurately.

A common example of how technology can facilitate meetings that wouldotherwise not be possible to conduct in person is to conduct a meetingvia satellite (such as on a television news program) or over theInternet (such as in a video conferencing context). Video conferencingcan enhance collaboration and allow participants to connect with oneanother on a personal level, without requiring the participants to bepresent in the same location. With computer systems equipped withdigital modems, digital video cameras, microphones, speakers, and thelike, users at different locations can participate in a video conferencein which conference participants can see and hear each other as theydiscuss various topics. In theory, video conferencing can providesignificant cost and time savings for conference participants whencompared with traditional meetings, and can allow collaboration betweenindividuals who would not be able to meet in person due to schedulingconflicts or travel restrictions.

However, such meetings still have many drawbacks, independent of thetraditional requirement that participants be in the same location. Forexample, live meetings are subject to time constraints. Even if theparticipants are not required to be in the same location, they mustagree to participate on a particular day, at a particular time, and fora particular length of time. Although the burden of travel may bereduced, other factors may still prevent participation by one or moreparties at the agreed-upon time. If a participant joins the meeting lateor leaves the meeting early, or if early parts of the discussion takelonger than expected, some topics may need to be allotted less time oromitted entirely. As another example, a live meeting requires anuninterrupted communication channel. A loss of connectivity betweenparticipants, even for a short time, can severely disrupt the meetingand reduce the available time in which the meeting can be conducted.These drawbacks are multiplied when more than two people areparticipating. Existing video conference tools can be used to conductvideo interviews but are typically focused on providing approximationsof in-person conversations, which tend to be disorganized, difficult toschedule, and difficult to analyze objectively. As a result, videoconferences have many of the same limitations that in-personconversations do.

SUMMARY

This summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This summary is not intended to identify key features ofthe claimed subject matter, nor is it intended to be used as an aid indetermining the scope of the claimed subject matter.

In one aspect, a server receives a plurality of pre-recorded videoprompts and generates an interview script comprising at least two of theplurality of pre-recorded video prompts according to a prompt-orderingalgorithm. Upon receiving an initiation request for an asynchronousvideo interview from the client computing device, the server transmits avideo prompt from the interview script to be displayed at a clientcomputing device as part of the asynchronous video interview, andreceives a streamed video response to the video prompt from the clientcomputing device. The interview script comprises the plurality ofpre-recorded video prompts. The server can perform algorithmic analysison content of the video response. The algorithmic analysis can includeapplying audio or video analysis algorithms to audio data or video data,respectively, of the video response. The algorithmic analysis also caninclude determining a time duration of the response. The analysisfacilitates determining a response score for the video response.

In another aspect, a server obtains response preference data indicatinga timing parameter for a video response in an asynchronous videointerview. The timing parameter can include, for example, a time limitfor the response, or an indication of whether recording of the responsebegins immediately after playback of the video prompt or after apredetermined time delay. The server transmits a video prompt andreceives a video response to the video prompt in accordance with theresponse preference data. For example, the video response may be limitedto a time limit. As another example, the video response may be recordedand then re-recorded, depending on preferences indicated in the responsepreference data.

In another aspect, a server computer in communication with a clientcomputing device comprises a processor and computer-readable storagemedia having stored thereon computer-executable instructions configuredto cause the server computer to transmit response preference dataconfigured to indicate a timing parameter (e.g., a preferred timeduration) for a video response to a video prompt in an asynchronousvideo interview; transmit the video prompt to be displayed in aninterviewee user interface presented at the client computing device;receive the video response from the client computing device, wherein thevideo response is recorded in accordance with the response preferencedata; and perform algorithmic analysis on the content of the videoresponse, wherein the algorithmic analysis facilitates determining aresponse score for the video response.

In another aspect, a computer-readable storage medium includescomputer-executable instructions configured to cause a client computingdevice to: determine a timing parameter for a video response to a videoprompt in an asynchronous video interview; and display a user interfaceconfigured to play the video prompt at the client computing device anddisplay a set of one or more user interface elements based at least inpart on the timing parameter. For example, if the timing parameterindicates that recording of the response will begin immediatelyfollowing playback of the video prompt, the set of user interfaceelements may include some elements that are compatible with the timingparameter (such as a countdown timer or a button that can be activatedto indicate that the response is complete) while omitting other elementsthat are not compatible with the timing parameter (such as a “beginrecording” button).

In another aspect, a video prompt and an information supplement (e.g., anews item) that relates to the content of the video prompt aretransmitted, and a video response to the video prompt and theinformation supplement is received. The information supplement relatesto content of the video prompt, and can be used to augment an interviewquestion.

In another aspect, a server performs analysis of a video responserecorded at a client computing device in a video interview. Based on theanalysis, the server automatically selects a video prompt (e.g., afollow-up question) to be displayed at the client computing device. Forexample, a follow-up prompt (e.g., “Please elaborate” or “Would youplease expand on that?”) can be automatically selected if a response toa previous question has a time duration that is below a threshold value.As another example, a text transcript or keywords from a response to aprevious question can be analyzed, and a follow-up question can beautomatically selected based on the analysis of the text transcript orkeywords.

In another aspect, a server receives information about an interviewee(e.g., personal information, demographic information, languagepreferences, etc.), performs analysis of the received information aboutthe interviewee, and, based on the analysis, automatically selects avideo prompt to be displayed at a client computing device as part of avideo interview. For example, if the information about the intervieweeincludes a language preference, a question that is presented in theinterviewee's preferred language can be automatically selected.

DESCRIPTION OF THE DRAWINGS

The foregoing aspects and many of the attendant advantages of thisinvention will become more readily appreciated as the same become betterunderstood by reference to the following detailed description, whentaken in conjunction with the accompanying drawings, wherein:

FIG. 1 is a block diagram that illustrates a generalized video interviewsystem according to various aspects of the present disclosure;

FIG. 2 is a block diagram that illustrates another example of a videointerview system according to various aspects of the present disclosure;

FIG. 3 is a block diagram that illustrates an example of an applicationsystem with video interview functionality according to various aspectsof the present disclosure;

FIGS. 4A, 4B, 5, 6, and 7 illustrate embodiments of an interviewee userinterface according to various aspects of the present disclosure;

FIG. 8 illustrates one embodiment of a reviewer user interface accordingto various aspects of the present disclosure;

FIGS. 9, 10, 11, 12, 13, and 14 are flow charts that illustrate exampletechniques according to various aspects of the present disclosure; and

FIG. 15 illustrates aspects of an exemplary hardware architecture of acomputing device suitable for use with various embodiments of thepresent disclosure.

DETAILED DESCRIPTION

The present disclosure includes descriptions of various aspects ofexemplary video interview systems and processes. In one examplescenario, an interviewee operating a computing device is prompted toplay one or more video prompts (e.g., questions or statements by aninterviewer intended to elicit a responses) on multimedia output devicesand record responses using multimedia input devices, as described inmore detail below. An interview engine then transmits (e.g., over anetwork such as the Internet) the interviewee's responses, which canthen be reviewed and analyzed.

Exemplary asynchronous video interview systems and processes describedherein do not require interviewees to participate in an interview in aparticular location, on a particular day, or at a particular time.Exemplary video interview systems and processes described herein also donot require live participation by interviewers, and facilitate enhancedreview of responses at a time that is convenient for the reviewers.Although it is a common view that an interviewer's present-senseevaluation of an interviewee's performance during a live interview is agood predictor of future performance, recent research supports the ideathat live interviews often fail to achieve their goals of accuratelyevaluating interviewees. See Daniel Kahneman, Thinking, Fast and Slow(2011).

Described video interview systems and related processes may be usefulfor entities that receive large numbers of applicants for a small numberof openings (e.g., universities, employers, or other institutions). Forexample, universities may receive thousands of applications, even thoughthey will admit only a small percentage of applicants. Using a videointerview system, an admissions officer of a university can browse andreview video interviews when making admissions decisions. As anotherexample, employers may receive hundreds of applications for a singleopen position. Using a video interview system, a human resources orrecruitment officer of an organization can browse and review videointerviews when making hiring decisions.

FIG. 1 is a block diagram that illustrates a generalized video interviewsystem 100 according to various aspects of the present disclosure. Thevideo interview system 100 includes a client device 102, an interviewserver 106, and an administrator device 108. The components of the videointerview system 100 may communicate with each other via a network 90.The network 90 may comprise one or more sub-networks (not shown). Forexample, the network 90 may comprise a local area network (e.g., a Wi-Finetwork) that provides access to a wide-area network such as theInternet. The client device 102 may be a computing device operated by anend user (e.g., an interviewee) to transmit and receive video data,audio data, or other data to the interview server 106. A reviewing user(or reviewer) operating the interview administrator device 108 mayconnect to the interview server 106 to, for example, upload recordedinterview questions, stream live interview questions, monitor liveinterview responses, and browse and review recorded interview responses.For example, recorded responses can be presented as a playlist thatallows reviewers to review responses in any order.

FIG. 2 is a block diagram that illustrates another example of a videointerview system. As shown in FIG. 2, the video interview system 200comprises a client device 202, an interview server 206, and anadministrator device 208.

In the example shown in FIG. 2, the interview server 206 comprises aninterview data store 220 and an interview management engine 222. Theinterview data store 220 stores data (e.g., video data, audio data,configuration information, or other data) that relates to videointerviews, such as data submitted by end users (e.g., an intervieweethat interacts with client device 202, an administrator that interactswith administrator device 208, etc.), as will be described furtherbelow. The interview management engine 222 interacts with the interviewdata store 220 and facilitates communication between the interviewserver 206 and other devices. The interview management engine 222 mayinclude a media server (e.g., an Adobe® Flash® Media Server or someother media server) that receives and transmits published media streamsor other formats of media data.

The interview data store 220 can store definitions that define elementsto be displayed to an end user on a client device 202 or administratordevice 208. The interview management engine 222 can use such definitionsto present graphical user interfaces to users. For example, a definitioncan be used to present a graphical user interface to guide aninterviewee to respond to video prompts such as questions or otherstatements intended to elicit a response. As another example, adefinition can be used to present a graphical user interface to guide areviewer to select a particular interview response for viewing, to guidean interviewer to record an interview question, or to perform othertasks. Definitions can include information defining a set of interfaceelements. The interface elements, such as text boxes, soft buttons,checkboxes, drop-down boxes, multimedia response interface elements,and/or the like, may receive input from the end user (e.g., in responseto prompts). The definition also can include information defining thelayout, appearance, and behavior of interface elements. Examples of userinterface layouts and elements are described in further detail below.

In the example shown in FIG. 2, the client device 202 includesmultimedia output device(s) 210, multimedia input device(s) 212, and aclient interview engine 214. The client interview engine 214 isconfigured to request information (e.g., user interface information,media data corresponding to interview questions) from the interviewserver 206 and send information (e.g., media data corresponding tointerview answers, authentication information, user preferences, orother information) to the interview server 206. The client interviewengine 214 is configured to cause multimedia output device(s) 210 tooutput content related to video interviews. For example, a displaydevice can display graphics and/or text corresponding to a userinterface, video corresponding to video prompts or previews of recordedresponses, or other content. In one example scenario, an interviewee isprompted to play an interview question and record a response usingmultimedia input device(s) 212, as described in more detail below. Theclient interview engine 214 then transmits the interviewee's responsesto the interview management engine 222 executing on interview server206. The interview management engine 222 can transmit the responses atan appropriate time (e.g., in response to an authorized request) toanother device such as administrator device 208.

In the example shown in FIG. 2, the administrator device 208 includesmultimedia output device(s) 230, multimedia input device(s) 232, and anadministrator interview engine 234. The administrator interview engine234 is configured to request information (e.g., user interfaceinformation, media data corresponding to interview answers) from theinterview server 206 and send information (e.g., media datacorresponding to interview questions, authentication information,configuration information, or other information) to the interview server206. The administrator interview engine 234 is configured to causemultimedia output device(s) 230 to output content related to videointerviews. For example, a display device can display graphics and/ortext corresponding to a user interface, video corresponding to recordedresponses, or other content. In one example scenario, a reviewer ispresented with a playlist of recorded responses by differentinterviewees, and can select a response to review. In response to theselection, the interview management engine 222 transmits the recordedresponse to the administrator interview engine 234. In another examplescenario, an interviewer is prompted to record a video prompt usingmultimedia input devices 232. The administrator interview engine 234then transmits the recorded video prompt to the interview managementengine 222. The interview management engine 222 can transmit the videoprompt at an appropriate time (e.g., in response to an authorizedrequest) to another device such as client device 202.

The client interview engine 214 and the administrator interview engine234 can be implemented in whole or in part by a web browserappropriately configured for conducting or participating in a videointerview, such as the Internet Explorer® browser by MicrosoftCorporation, the Firefox® browser by the Mozilla Foundation, and/or thelike. Configuration of a web browser may include browser plug-ins orother modules that facilitate recording and viewing video.Alternatively, the client interview engine 214 can be implemented as acustom desktop application or mobile application specially configuredfor conducting or participating in a video interview.

In any of the described examples, an “engine” may include computerprogram code configured to cause one or more computing device(s) toperform actions described herein as being associated with the engine.For example, a computing device can be specifically programmed toperform the actions by having installed therein a tangiblecomputer-readable medium having computer-executable instructions storedthereon that, when executed by one or more processors of the computingdevice, cause the computing device to perform the actions. An exemplarycomputing device is described further below with reference to FIG. 15.The particular engines described herein are included for ease ofdiscussion, but many alternatives are possible. For example, actionsdescribed herein as associated with two or more engines on multipledevices may be performed by a single engine. As another example, actionsdescribed herein as associated with a single engine may be performed bytwo or more engines on the same device or on multiple devices.

In any of the described examples, a “data store” contains data asdescribed herein and may be hosted, for example, by a high-capacitydatabase management system (DBMS) to allow a high level of datathroughput between the data store and other components of a videointerview system. The DBMS may also allow the data store to be reliablybacked up and to maintain a high level of availability. For example, adata store may be accessed by other components of a video interviewsystem via a network, such as a private network in the vicinity of thesystem, a secured transmission channel over the public Internet, acombination of private and public networks, and the like. Instead of orin addition to a DBMS, a data store may include structured data storedas files in a traditional file system. Data stores may reside oncomputing devices that are part of or separate from components of videointerview systems described herein. Separate data stores may be combinedinto a single data store, or a single data store may be split into twoor more separate data stores.

In any of the described examples, media data can be captured bymultimedia input devices and transmitted or stored for futureprocessing. The processing may include encoding data streams, which canbe subsequently decoded for presentation by multimedia output devices.Captured media data can be stored by saving media data streams as fileson a computer-readable storage medium (e.g., in memory or persistentstorage on client device 202, interview server 206, administrator device208, or some other device). Referring to FIG. 2, multimedia inputdevices 212, 232 may include a video camera. A video camera, whenactive, may provide a stream of video data. As another example,multimedia input devices 212, 232 may include a microphone. Amicrophone, when active, may provide a stream of audio data. Multimediainput devices 212, 232 can be separate from and communicatively coupledto the client device 202 or administrator device 208, or can be integralcomponents of the client device 202 or the administrator device 208,respectively. In some embodiments, multiple multimedia input devices maybe combined into a single device (e.g., a video camera with anintegrated microphone). Any suitable multimedia input device eithercurrently known or developed in the future may be used with describedvideo interview systems.

The multimedia output devices 210, 230 may include video output devicessuch as a display or touchscreen. The multimedia output devices 210, 230also may include audio output devices such as external speakers orearphones. The multimedia output devices 210, 230 can be separate fromand communicatively coupled to the client device 202 or administratordevice 208, or can be integral components of the client device 202 orthe administrator device 208, respectively. In some embodiments,multiple multimedia output devices may be combined into a single device(e.g., a display with built-in speakers). Any suitable multimedia outputdevice either currently known or developed in the future may be usedwith described video interview systems.

In any of the described examples, digital signal processors (which canbe implemented in hardware, software, or some combination of hardwareand software) can be used for processing media data such as audio dataand video data. For example, a digital signal processing module caninclude encoders to encode and/or decoders to decode encoded data informats such as MP3, Vorbis, AAC, HE-AAC, or Windows Media Audio (WMA)for audio, or MPEG-2/H.262, H.263, VC-1, or H.264 for video. Differentencoding and decoding modules are typically used for encoding anddecoding data in different formats. In a typical scenario, an encoder onone device encodes media data for subsequent decoding by a decoder onanother device, although a single device can include both encoders anddecoders. For example, the client device 202 may include one or moredecoders for decoding encoded media data (e.g., video data thatcorresponds to interview questions), as well as one or more encoders forencoding captured media data (e.g., video data that corresponds tointerview answers).

Encoded media data can be delivered in a data stream. As used herein,the term “stream” refers generally to a flow of information deliveredfrom one device to another over a communication link (e.g., a networkconnection), and is not limited to any particular content, transferprotocol, or data format. A stream may represent portions or “packets”of a larger file, such an audio file or a video file. In general, astream can be used for different purposes, such as delivering livecontent (e.g., live audio or video broadcasts) or for deliveringpre-recorded content without having to deliver a larger file. A streamcan be processed for playback on a device. For example, a data streamcomprising audio data and video data can be processed by decoding theaudio data and the video data and rendering it for output by one or moreoutput devices. The term “streaming” can be used to refer to the processof delivering or receiving a stream. In a typical streaming mediascenario, media information is buffered by the receiving device beforebeing processed for playback in order to mitigate the possible effectsof varying delivery rates. Streaming can be accomplished using anyprotocols suitable for streaming media such as Real-Time TransportProtocol (RTP), Real-Time Streaming Protocol (RTSP), Real-Time ControlProtocol (RTCP), other protocols, or combinations of protocols (e.g.,combinations of transfer protocols and control protocols). In general,data that can be delivered via streaming also can be packaged intolarger files for delivery. For example, a client device can download afile containing video data corresponding to an entire pre-recordedinterview question and can begin decoding and playing the correspondingvideo when the entire video file has been downloaded.

A video interview process may depend on conditions of components of thevideo interview system 200. Conditions can be monitored to help ensureproper functioning of the interview system 200. For example, a componentof the system 200 (e.g., interview management engine 222) may testcomponents of the client device 202 (e.g., multimedia input devices 212such as a video camera and microphone) or cause the client device 202 totest its own components before allowing the interview process to beginor before performing certain actions, such as recording responses. Asanother example, a component of the system 200 may check microphonelevels, lighting levels, or other parameters to make sure that recordedanswers will be readily audible and viewable. To check such parameters,a test question and answer can be used. The test answer can be analyzedby a component of the system 200 (e.g., interview management engine 222)or by a human user (e.g., a reviewer operating administrator device 208or an interviewee operating client device 202). Adjustments can then bemade based on the detected conditions. Upon receiving an indication of asuccessful test, the interview process can continue.

Depending on detected conditions, users may be prompted to makeadjustments. For example, if no audio signal is detected, or if theaudio signal is too weak, too strong, or distorted, an interviewee canbe prompted to check the function or positioning of a microphone at theclient device 202. As another example, if no video signal is detected,the interviewee can be prompted to check the function of the videocamera. As another example, if the video signal is too dark, theinterviewee can be prompted to turn on or move closer to a light source.As another example, if the interviewee is not visible in the frame, theinterviewee can be prompted to adjust the positioning of the videocamera or to sit or stand in a different location.

The video interview process can be conducted in different ways. Forexample, an interviewee can be asked a predetermined set of questions(e.g., in a predetermined order or in random order) or a random set ofquestions selected from a question bank. The interviewee also can beasked to select particular questions from a set of questions. Thequestions that are asked or available for selection by the intervieweecan be the same for all interviewees or can vary among interviewees. Or,some questions can be asked of all interviewees, while other questionsare targeted to particular interviewees. Whichever questions are used,recording the questions ahead of time can help to reduce the risk ofinterviewer bias by ensuring that interviewees responding to aparticular question will have that question presented to them in thesame way.

Application forms or other data related to interviewees can be analyzedto determine whether particular subjects may be explored by askingparticular questions, or whether some questions that are applicable tosome interviewees may not be applicable to others. Targeted questionscan be selected algorithmically based on data associated with aninterviewee (e.g., geographic location, native language, gender,credentials such as grade point average, or other data). For example, aquestion selection algorithm can determine that the interviewee hasindicated salary requirements that are higher than an open position willpay and select a question that asks whether the interviewee would bewilling to accept a lower salary. As another example, a questionselection algorithm can determine that the interviewee has claimedfinancial hardship when applying for a scholarship and select a questionthat asks the interviewee to describe the relevant circumstances.Targeted questions also can be based on previous interview questions oranswers. For example, if an interviewee has elected to answer a questionabout a particular achievement in a previous job, a question selectionalgorithm can select a follow-up question that asks how theinterviewee's employer benefited from that achievement.

Video prompts can be presented in different ways. For example, aninterviewee can set question preferences such as language preferences(e.g., the interviewee's native language) or preferences for howquestions are presented (e.g., whether questions will be asked by a maleor female interviewer). Question preferences also can be setautomatically or by a user affiliated with the organization that isconducting the interview (e.g., based on facts that are known about aninterviewee). For example, a female interviewee may be automaticallypresented with questions that are asked by a female interviewer.Preferences can be the same for all questions, or can vary by question.For example, a native Spanish speaker applying to a university wherecourses are taught in English may have some questions presented inSpanish while also having some questions presented in English in orderto demonstrate command of the English language. Live questions can beused in place of or in combination with recorded questions. Though theymay be presented to an interviewee without being prerecorded, livequestions can be recorded and stored (e.g., for later playback toreviewers).

Interview responses can be recorded and collected in different ways. Forexample, responses can be collected and stored in a single video file inseparate files. Separate files can be combined into larger filescomprising multiple responses. Questions also can be stored in the samefiles as answers, allowing for the possibility that an entire videointerview can be stored in a single file. During the answering process,a client device can display a video or still-frame image of aninterviewer appearing to listen to the interviewee's responses. Optionsfor interview responses can be configured. For example, an administratorcan determine whether an interviewee can play back a recorded answer asa preview and/or re-record the answer before submitting it. Disablingre-recording can encourage candid, spontaneous answers, while allowingre-recording can make applicants more comfortable with the interviewprocess and potentially encourage more applicants to agree to theinterview.

The timing of responses can be handled in different ways. For example,recording of an answer can be configured to start when playback of aninterview question stops. However, other approaches can be used. Forexample, a time delay between the end of a question and recording of theanswer can be introduced to allow the interviewee time to think aboutthe question before answering. Time delays between playback of questionsand recording of answers can be configured (e.g., via administratordevice 208). For example, the next question (if further questions are tobe asked) can begin playback immediately upon detection of a completedanswer, or after a delay. A completed response can be detected indifferent ways. For example, a response can be considered complete whenan extended period of silence is detected following an extended periodof speech, when a particular phrase (e.g., “next question” or “finalanswer”) is detected using speech recognition techniques, or when a userinterface element (e.g., a “stop recording” or “next question” button)is activated. Options for detecting completion of an answer can beconfigured as desired. For example, an administrator can choose how todetect when an answer is complete. The interviewee also may be givencontrol over when the recording of an answer will begin (e.g., via a“record” button or other user interface element that can be activated bya user as desired).

Information can be collected to provide a description (such as atimeline) for the interview. Responses can be associated with dataobjects such as time points. Time points can indicate, for example, whenquestions were asked and answered. As another example, particularquestions and answers can be identified (e.g., by number, by content ofthe question or answer, or in some other way). Timestamps and otherinformation can be added to the timeline automatically or by users. Forexample, a user of administrator device 208 can be given the option toadd comments to the timeline at selected points to flag noteworthyevents (e.g., a particularly insightful comment or a defensive reactionby the interviewee) or invite discussion among the other reviewers.

Algorithmic Analysis Examples

Algorithmic analysis can be performed on interview responses.Algorithmic analysis can facilitate evaluation of interview responses,such as by providing information on which a response score can be based.Different combinations of algorithmic analysis results can be used togenerate a score, and can be weighted to a greater or lesser extent,depending on system design, reviewer preference, or other factors. Otherinformation can be used to generate a response score (e.g., incombination with algorithmic analysis results). For example, a reviewercan evaluate a response (e.g., by viewing a response and looking atalgorithmic analysis results) and provide his or her own score for theresponse. The score provided by the reviewer can be used as a componentscore on which the response can be based. For example, component scoresfrom several reviewers can be averaged and combined with other componentscores (e.g., component scores based on algorithmic analysis results) toarrive at an overall response score.

Algorithmic analysis also can be used for other purposes. For example,algorithmic analysis can be used to determine whether a follow-upquestion should be asked, and what the follow-up question should be. Thespecific types of algorithmic analysis described herein are onlyexamples, and can be replaced or supplemented with other types ofalgorithmic analysis, as may be appropriate for a particular system orprocess to be implemented.

Algorithmic analysis can be performed while an interview is beingconducted (e.g., to determine whether a property of the answer indicatesthat a follow-up question might be appropriate). For example, todetermine whether a response has a time duration that is shorter thanexpected or desired, the time duration of a response can be measured (orobtained in some other way, such as by inspecting metadata correspondingto the response) and compared with a threshold value. If the timeduration is below the threshold value, a follow-up question (e.g.,“Would you like to expand on that?” or “Can you tell me a bit more aboutthat?”) can be played to prompt the interviewee to further elaborate onhis or her answer. If an answer to a question is determined to haveanother property, such as low volume, a follow-up question (e.g., “I'mhaving trouble hearing you—would you mind repeating your answer?”) canbe played to prompt the interviewee to speak in a louder voice or adjustthe microphone.

During or after the interview, speech recognition algorithms can be usedto obtain a text transcript of some or all of the interview. Furtheranalysis can be performed on the text transcript (e.g., by algorithms orby human reviewers). For example, a score can be applied to answersbased on the content of the text transcript (e.g., based on keywords),and scores for different interviewees can be compared. As anotherexample, follow-up questions can be selected based on content (e.g.,keywords) of the text transcript.

During or after the interview, algorithms can be used to analyze audioand video information to detect possible emotional responses orbehavioral cues. For example, facial recognition or other computervision algorithms can be used to determine level of eye contact,identify facial expressions, interpret body language, or perform otherkinds of analysis. As another example, various algorithms can be used toanalyze audio waveform peaks, inflection range, pauses and gaps (e.g.,gap from the end of a question to the beginning of an answer), or othercharacteristics in order to provide information about an interviewee'sanswer. Audio and video analysis algorithms can be used to evaluateresponses. For example, if an interviewee is applying for a stressfuljob that requires calm responses in emotional situations, audio or videoanalysis algorithms can be used to detect emotion or stress during ananswer to a particular question.

Many examples of algorithmic analysis described herein can be used togenerate a score for an interview response, either independently or incombination. In one example scenario, an applicant for a position thatrequires frequent and extensive conversations with customers may receivea score based on algorithmic analysis results such as average length ofresponse (e.g., by receiving a lower score for short responses toopen-ended questions). Further, some examples of algorithmic analysiscan be used both to generate a score and for other purposes. In theexample scenario described above, the applicant may receive a lowerscore for short responses to open-ended questions, and the shortresponses may also automatically trigger follow-up questions.

Review of Responses

Any number of reviewers can review video interviews, although access canbe limited as desired (e.g., by limiting permissions to only selectedstaff members). Reviewers may use information collected during theinterview (such as answers to questions) or derived from the interview(such as text analytics) to generate other information, such as a scorefor the interview. Such information can be correlated with otherinformation that relates to the interviewee, such as an ultimatedecision on the interviewee (e.g., whether they are admitted to auniversity or hired after the interview), future performance of theinterviewee (e.g., grade point average or work review scores for someperiod of time (e.g., one year) after admission or hiring,respectively), or other information. Reviewers can associate performancescores with individual answers and can give an overall score to aninterviewee based on the overall interview and/or other information,such as information contained in a resume or application form.

Reviewers can be given the option to record follow-up questions whichcan then be presented to the interviewee (or a group of interviewees) ina second video interview, which may be similar in purpose to anin-person follow-up or “call-back” interview. Personalized follow-upquestions can be based on previous responses or other informationprovided by an interviewee. Personalized follow-up questions can becombined with other questions that may not be personalized. For example,some questions may be presented to all interviewees that advance to asecond round of interviews, while other questions in the second round ofinterviews may be personalized to individual interviewees.

Reviewers also can record their own analysis (e.g., by making a video oraudio recording) of any of the interviewee's responses. The recordedanalysis can be saved for future reference or analyzed by otherreviewers.

Different reviewers can review different parts of an interview. Forexample, in an interview for university admission, questions relating toa particular area of study can be designated for review by faculty inthe relevant department, while questions of more general interest can bereviewed by other reviewers. As another example, reviewers may choose toreview individual components (e.g., audio only, video only, audio andvideo) of an interview or information related to an interviewee (e.g.,information from a resume or application form). Reviewers may reviewvideo interviews based on different criteria. For example, in aninterview for a job that requires frequent public appearances, somereviewers may evaluate video interviews based on criteria such aspersonal appearance and speaking style, while other reviewers may focuson the content of the interviewee's answers. Evaluation can be performedon an interview that is in progress or an interview that has beencompleted.

Reviewers can be given the option of providing feedback on interviews.For example, the administrator device 208 can include a user interfacewith graphical dials, sliders, buttons, or other controls to allowreviewers to provide feedback during a review of the interviewee'sresponses. Feedback can be aggregated into an overall score. Forexample, when a reviewer provides feedback (such as a positive ornegative rating), a new score can be stored along with a timestamp orcode linking the feedback to a portion of the video response. Suchfeedback can be given as the interviewee is providing a live response,or after the response is recorded. Aggregation of feedback provided bymultiple reviewers can allow for identification of trends to helpevaluate interviews (e.g., by identifying an answer or a portion ofanswer that was particularly liked or disliked by several reviewers).

During playback of the video interview files, trick modes such as fastforward, rewind, pause, and various other playback options can be used.For example, reviewers can pause playback to enter comments or otherannotations.

Combining Video Interview Functionality with Other Tools

Described video interview systems (or components of such systems) andprocesses may be used in combination with other tools. Described videointerview systems can be integrated into existing workflows as part ofan overall recruitment process. For example, systems that provideservices such as application form processing can be adapted to includevideo interview functionality.

In the example shown in FIG. 3, an application system 300 with videointerview capability includes a form processing server 304 that maycause an information form (such as a fillable application form) to bedisplayed on a client device 302. Such forms can be used to collectdemographic information that can be used to analyze an applicant pool,or personal information that can be used to evaluate individualapplicants. The form can include standard text prompts and answerfields, and also can include multimedia prompts and/or answer fields.The form also can include information relating to a video interview tobe conducted as part of the application process. Such information caninclude a URL or hyperlink that allows a user to connect to theinterview server 306 and participate in a video interview. The formprocessing server 304 may store information entered into a form.

The client device 302 can include a form display engine (not shown)configured to request form definition information, along with componentsshown in client device 202 in FIG. 2. Upon receiving the form definitioninformation, the form display engine can display form prompts and inputfields to an end user and collect the end user's responses in the inputfields. The responses can then be sent to one or more other devices forprocessing. The administrator device 308 can include tools for browsingand reviewing form submissions, along with components shown inadministrator device 208 in FIG. 2. The form display engine and toolsfor browsing and reviewing form submissions can be implemented in astandard web browser, such as the Internet Explorer® browser byMicrosoft Corporation, the Firefox® browser by the Mozilla Foundation,and/or the like. Alternatively, the form display engine and tools forbrowsing and reviewing form submissions can be implemented in aspecially-configured desktop application or mobile application.

Once form submissions are received and stored by the form processingserver 304, a reviewer may use the administrator device 308 to browseand review the form submissions, which may be stored by the formprocessing server 304. For example, a reviewer may review formsubmissions as part of a review process that also includes reviewingvideo interview answers on the administrator device 308. Review of formsubmissions may be useful, for example, where a reviewer wishes toconfirm the accuracy of an applicant's answer in an interview bycomparing the answer with information in a form submission. Formsubmission information may include a transaction ID, a record of theinformation submitted, and a record of the prompts that were displayed.The form submission information may be associated with interview data(e.g., via a transaction ID) to facilitate organization and review.

User Interface Examples

FIGS. 4A, 4B, 5, 6, and 7 illustrate example user interfaces that may bepresented to an interviewee or other user of a client device (e.g.,client device 202 in FIG. 2), and FIG. 8 illustrates an exampleinterface that may be presented to a reviewer or other user of anadministrator device (e.g., administrator device 208 in FIG. 2). Theillustrated interfaces are depicted as being presented in a web browser401. The interface elements displayed by the web browser 401 aregenerated by an interview engine (e.g., client device interview engine214 or administrator device interview engine 234, respectively), whichmay include code downloaded by the web browser 401 from the interviewserver 206.

The interfaces shown in FIGS. 4A through 8 are exemplary only, and notlimiting. The elements shown in FIGS. 4A through 8 may be supplementedor replaced by any number of other elements exhibiting the samefunctionality or different functionality. The interfaces shown in FIGS.4A through 8 may be provided in a browser 401 as shown, in a standaloneapplication executing on a client device or administrator device, in aserver application executing in a cloud computing environment, or insome other way. In any of the described examples, user interfaceelements can be actuated by a keystroke, voice activation, mouse click,or any other suitable user input event. User input events such as voiceactivation that do not require a user to manipulate an input device(such as a mouse or keyboard) may enable an interviewer or aninterviewee to record a more natural looking video.

In the example shown in FIG. 4A, an interviewee interface 400 includes avideo element 412 depicting an interviewer 414 and interactive userinterface elements including a “repeat question” button 416 and a recordbutton 418, which can be used to respond to a question identified as“Question 1” by a question identifier 422. In the example shown in FIG.4A, the text of the question (“Where do you see yourself in ten years?”)is shown for illustrative purposes only and need not be displayed.Typically, the question is spoken by the interviewer 414 and presentedwith accompanying audio, although this is not required and may not bedesirable in some scenarios such as where the interviewee ishearing-impaired. In such scenarios, the text of the question can bedisplayed as shown or communicated in some other way (e.g., by signlanguage, with or without subtitles). As illustrated, the video element412 can respond to activation of the record button 418, at which pointthe client device 202 will begin recording the interviewee's answer. Therepeat question button 416 can be used to cause the question to playagain. In the example shown in FIG. 4B, an interviewee interface 450contains many of the same elements as interface 400, but some userinterface elements (e.g., “repeat question” button 416, record button418) are omitted. The interface 450 can be useful in a scenario whererecording of the interviewee's response begins immediately after a videoprompt has stopped playing. Such scenarios can better mimic a liveinterview by omitting user options for waiting indefinitely beforeanswering a question or for previewing a recorded answer beforesubmitting it.

In the example shown in FIG. 5, an interviewee interface 500 includes avideo element 512 depicting the interviewee 504 (e.g., as viewed by avideo camera attached to the client device 202) and interactive userinterface elements including a record button 516 and a stop button 518,which can be used to respond to a question identified as “Question 1” bya question identifier 522. The text of the question (“Where do you seeyourself in ten years?”) can be provided as a reminder, but need not bedisplayed. As illustrated, the client device can respond to activationof the record button 516, at which point the client device will beginrecording the interviewee's answer. The stop button 518 can be used tostop recording. Preview text 520 indicates that a preview of therecorded answer will be available after recording. The countdown timer524 indicates time remaining for recording an answer. A status indicator526 indicates that the video shown in the video element 512 is live, andnot previously recorded.

In the example shown in FIG. 6, an interviewee interface 600 includes avideo element 612 depicting the interviewee 504 in an answer previewmode. Interactive user interface elements including a play button 616, acancel button 628, and a save button 618 can be used to play a recordedanswer, cancel a recorded answer, and save a recorded answer,respectively. A status indicator 626 indicates that the video shown inthe video element 612 is previously recorded.

In the example shown in FIG. 7, an interviewee interface 700 includesmultiple video elements 712, 732. The video element 712 depicts theinterviewee 504 and includes interactive user interface elementsincluding a record button 516 and a stop button 518. The countdown timer524 indicates time remaining for recording an answer. The video element732 is configured to show a recorded answer in an answer preview mode.Preview text 720 indicates that a preview of the recorded answer will beavailable after recording.

In the example shown in FIG. 8, a reviewer interface 800 includes avideo element 812 depicting the interviewee 504 (identified as“Applicant 1”) in a recorded response. Status indicator 826 indicatesthat the video shown in the video element 812 is a previously recordedresponse to a question identified as “Question 1” by a questionidentifier 822. Playback controls 830 allow playback of the response tobe paused, rewound, etc. during review. The feedback button 840 is anelement that a reviewer can activate to give feedback on the response.The timeline 850 shows a graphical depiction of the interview segmentbeing reviewed. The marker 860 shows the current playback position. Theflag 870 indicates a time point in the interview. The time pointassociated with the flag 870 can indicate, for example, a time at whicha reviewer added feedback, the beginning of an answer, or some otheraspect of the interview. The flag 870 may itself be an active element inthe user interface 800. For example, clicking on the flag 870 may causeinformation such as reviewer feedback to be displayed. A waveform 880provides a visual depiction of a signal (such as an audio signal)associated with the interview. The waveform 880 and/or other tools canbe used to analyze an interview (e.g., by showing points in theinterview where the interviewee is speaking louder or softer in responseto different questions).

Many alternatives to the user interfaces described above are possible.For example, in any of the examples described herein, multiple videoelements can be used. A reviewer can be presented with an interface thatshows the interviewee in one video element and the interviewer inanother video element. Or, an interface can show multiple intervieweesin multiple video elements (e.g., giving responses to the samequestion). Multiple video elements can be displayed simultaneously, orone after the other, in any orientation or configuration. As anotherexample, interviewees and reviewers can be presented with informationthat differs from the information shown in FIGS. 4A through 8. In afollow-up interview context, an interviewee can be provided with anindicator in the user interface that identifies the current interview asa follow-up interview, and a reviewer can be provided with informationthat indicates, for example, an interviewee's previous response to aquestion in a first round of interviews. Such information can include,for example, a transcript of the previous response or feedback on theprevious response that has been provided by a reviewer. Such informationcan be useful, for example, to help a reviewer to determine whether aninterviewee's response to a follow-up question is consistent with theinterviewee's previous response.

Example Techniques

FIGS. 9 through 14 illustrate example techniques that may be used whenconducting video interviews.

In the technique 900 shown in FIG. 9, at 910, a computing device (e.g.,a server) transmits a video prompt to be displayed at a client computingdevice as part of a video interview. At 920, the computing devicereceives a video response from the client computing device. In oneexample scenario, the video response includes video data and audio data.At 930, the computing device performs algorithmic analysis on content ofthe video response. The algorithmic analysis can include applying audioor video analysis algorithms to the audio data or the video data,respectively. The algorithmic analysis also can include determining atime duration of the response. The analysis facilitates determining aresponse score for the video response. The computing device can causethe video response and/or results of the algorithmic analysis to bedisplayed in a reviewer user interface. The computing device may receivefeedback or a component score from a reviewer (e.g., via the revieweruser interface) on which the response score can be based.

In the technique 1000 shown in FIG. 10, at 1010, a computing device(e.g., a server) obtains response preference data indicating a timingparameter for a video response in a video interview. The responsepreference data may be obtained, for example, from an administratordevice, or a data source such as a database. In one example scenario,the response preference data indicates preferences set by anadministrator for how a video interview should be conducted. Theresponse preference data may be transmitted to a client device at whichthe response will be recorded. Alternatively, response preference datamay be used at a server to enforce rules for recording the response at aclient device, without requiring transmission of response preferencedata directly to the client device.

The timing parameter can include a time limit for the response, or anindication of whether recording of the response begins immediately afterplayback of the video prompt, or after a predetermined time delay. Timedelays may be desirable to allow interviewees to consider their answersfor a limited time before answering. Besides timing parameters, responsepreference data also may indicate other preferences, such as whetherre-recording of a response is permitted. At 1020, the computing devicetransmits a video prompt. At 1030, the computing device receives a videoresponse to the video prompt in accordance with the response preferencedata. For example, the video response may be limited to a time limit. Asanother example, the video response may be recorded and thenre-recorded, depending on preferences indicated in the responsepreference data.

In the technique 1100 shown in FIG. 11, at 1110, a computing device(e.g., a server) transmits a video prompt to be displayed as part of avideo interview in an interviewee user interface presented at a clientcomputing device. At 1120, the computing device transmits an informationsupplement to be displayed at the client computing device. Theinformation supplement relates to content of the video prompt and can beused to augment an interview question.

In one example scenario, an applicant for a position (e.g., a politicaladvisor or public relations position) answers questions that areintended to evaluate the applicant's ability to quickly review a newsitem and develop a strategy for responding to the news item. In thisexample scenario, the information supplement can comprise a news item,such as a video clip, picture, news article, or newsfeed with dailyheadlines, that relates to the content of a question (e.g., “How wouldyou respond if a journalist showed you this article?” or “Which of theseheadlines merits the most attention, and why?”). In another examplescenario, the information supplement can comprise an information form(e.g., a web form) to be filled out by the interviewee by supplying userinput at the client computing device. In another example scenario, theinformation supplement can be a legal document (e.g., an exhibit) thatrelates to an interview question in a legal context (e.g., a depositionin which the exhibit is being discussed). The information supplement canbe presented to the interviewee at the same time as an interviewquestion, prior to an interview question, and/or after an interviewquestion.

In the technique 1200 shown in FIG. 12, at 1210, a computing device(e.g., a server) performs analysis of a video response recorded at aclient computing device in a video interview. At 1220, based on theanalysis, the computing device automatically selects a video prompt(e.g., a follow-up question) to be displayed at the client computingdevice. For example, a follow-up prompt (e.g., “Please elaborate” or“Would you please expand on that?”) can be automatically selected if aresponse to a previous question has a time duration that is below athreshold value. As another example, a text transcript or keywords froma response to a previous question can be analyzed, and a follow-upquestion can be automatically selected based on the analysis of the texttranscript or keywords. In one example scenario, if a text transcript ofa response in a job interview contains the word “salary,” a follow-upquestion that asks about the interviewee's salary requirements can beautomatically selected.

In the technique 1300 shown in FIG. 13, at 1110, a computing device(e.g., a server) receives interviewee information. Intervieweeinformation refers to information about the interviewee that does notcome directly from a previous video response in the video interview,although the same information may also be provided in a video response.For example, interviewee information can be received from theinterviewee via user input in a web form presented at a client computingdevice prior to the interview. Interviewee information also can bereceived from other sources, such as a database containing informationpreviously provided by the interviewee. The interviewee information caninclude personal information (e.g., age, name, residence address),demographic information, language preferences, etc. The intervieweeinformation also can include information that is relevant to the contextof a particular interview. For example, in a job interview, theinterviewee information can include salary history, education, or otherrelevant information. In an admissions interview for a university, theinterviewee information can include standardized test scores, academictranscript data, or other relevant information.

At 1320, the computing device performs analysis of the receivedinterviewee information. At 1330, based on the analysis, the computingdevice automatically selects a video prompt to be displayed at a clientcomputing device as part of a video interview. For example, if theinterviewee information includes details relating to a college degree, aquestion relating to the interviewee's college experience (e.g., “Whatwas the most important thing you learned in college?”) can beautomatically selected based on the interviewee information. As anotherexample, if the interviewee information includes a language preference,a question that is presented in the interviewee's preferred language canbe automatically selected.

In the technique 1400 shown in FIG. 14, at 1410, a computing device(e.g., a client computing device) determines a timing parameter (e.g., atime limit) for a video response to a video prompt in a video interview.For example, a server that provides the video prompt to be displayedalso may send configuration information that defines the timingparameter. At 1420, the computing device displays a user interfaceconfigured to play the video prompt and displays a set of user interfaceelements based at least in part on the timing parameter. For example, ifthe timing parameter indicates a time limit for the response or apredetermined time delay that precedes recording of the video response,a countdown timer can be displayed that shows the time remaining for theresponse or the time remaining before recording begins, as appropriate.As another example, if the timing parameter indicates that recording ofthe response will begin immediately following playback of the videoprompt, the set of user interface elements may include some elementsthat are compatible with the timing parameter (such as a countdown timeror a button that can be activated to indicate that the response iscomplete) while omitting other elements that are not compatible with thetiming parameter (such as a “begin recording” button). At 1430, thecomputing device records the video response in accordance with thetiming parameter. For example, if the timing parameter indicates a timelimit, recording can be stopped when the time limit is reached. Asanother example, if the timing parameter indicates a predetermined timedelay that precedes recording of the video response, recording can beginwhen the predetermined time delay has expired.

Many alternatives to the described techniques are possible. For example,processing stages in the various techniques can be separated intoadditional stages or combined into fewer stages. As another example,processing stages in the various techniques can be omitted orsupplemented with other techniques or processing stages. As anotherexample, processing stages that are described as occurring in aparticular order can instead occur in a different order. As anotherexample, processing stages that are described as being performed in aseries of steps may instead be handled in a parallel fashion, withmultiple modules or software processes concurrently handling one or moreof the illustrated processing stages. As another example, processingstages that are indicated as being performed by a particular device ormodule may instead be performed by one or more other devices or modules.

Operating Environment

In any of the examples described herein, client devices andadministrator devices may be any suitable computing devices, including,but not limited to, laptop computers, desktop computers, smart phones,tablet computers, and/or the like. Interview servers may includesuitable computing devices configured to provide services described infurther detail below. As used herein in the context of a server-clientrelationship, the term “server” refers generally to a computing devicethat provides information (e.g., video and audio data) and/or servicesto other devices over a communication link (e.g., a network connection),and is not limited to any particular device configuration. Servers mayinclude one or more suitable devices, such as dedicated server computingdevices, or virtualized computing instances or application objectsexecuting on a computing device. The term “client” can be used to referto a computing device (e.g., a client device 202, an administratordevice 208) that obtains information and/or accesses services providedby a server over a communication link, and is not limited to anyparticular device configuration. However, the designation of aparticular device as a client device does not necessarily imply orrequire the presence of a server. At various times, a single device mayact as a server, a client, a server and a client, or neither, dependingon context and configuration. Actual physical locations of clients andservers are not necessarily important, but the locations can bedescribed as “local” for a client and “remote” for a server toillustrate a common usage scenario in which a client is receivinginformation provided by a server at a remote location.

FIG. 15 is a block diagram that illustrates aspects of an exemplarycomputing device 1500 appropriate for use in accordance with embodimentsof the present disclosure. The description below is applicable toservers, personal computers, mobile phones, smart phones, tabletcomputers, embedded computing devices, and other currently available oryet-to-be-developed devices that may be used in accordance withembodiments of the present disclosure.

In its most basic configuration, the computing device 1500 includes atleast one processor 1502 and a system memory 1504 connected by acommunication bus 1506. Depending on the exact configuration and type ofdevice, the system memory 1504 may be volatile or nonvolatile memory,such as read only memory (“ROM”), random access memory (“RAM”), EEPROM,flash memory, or other memory technology. Those of ordinary skill in theart and others will recognize that system memory 1504 typically storesdata and/or program modules that are immediately accessible to and/orcurrently being operated on by the processor 1502. In this regard, theprocessor 1502 may serve as a computational center of the computingdevice 1500 by supporting the execution of instructions.

As further illustrated in FIG. 15, the computing device 1500 may includea network interface 1510 comprising one or more components forcommunicating with other devices over a network. Embodiments of thepresent disclosure may access basic services that utilize the networkinterface 1510 to perform communications using common network protocols.The network interface 1510 may also include a wireless network interfaceconfigured to communicate via one or more wireless communicationprotocols, such as WiFi, 2G, 3G, 4G, LTE, WiMAX, Bluetooth, and/or thelike.

In the exemplary embodiment depicted in FIG. 11, the computing device1500 also includes a storage medium 1508. However, services may beaccessed using a computing device that does not include means forpersisting data to a local storage medium. Therefore, the storage medium1508 depicted in FIG. 15 is optional. In any event, the storage medium1508 may be volatile or nonvolatile, removable or nonremovable,implemented using any technology capable of storing information such as,but not limited to, a hard drive, solid state drive, CD-ROM, DVD, orother disk storage, magnetic cassettes, magnetic tape, magnetic diskstorage, and/or the like.

As used herein, the term “computer-readable medium” includes volatileand non-volatile and removable and non-removable media implemented inany method or technology capable of storing information, such ascomputer-readable instructions, data structures, program modules, orother data. In this regard, the system memory 1504 and storage medium1508 depicted in FIG. 15 are examples of computer-readable media.

For ease of illustration and because it is not important for anunderstanding of the claimed subject matter, FIG. 15 does not show someof the typical components of many computing devices. In this regard, thecomputing device 1500 may include input devices such as a keyboard,keypad, mouse, trackball, microphone, touchpad, touchscreen, stylus,and/or the like. Such input devices may be coupled to the computingdevice 1500 by wired or wireless connections including RF, infrared,serial, parallel, Bluetooth, USB, or other suitable connectionsprotocols using wireless or physical connections. The computing device1500 may also include output devices such as a display, speakers,printer, etc.

In general, functionality of computing devices described herein may beimplemented in computing logic embodied in hardware or softwareinstructions, which can be written in a programming language, such as C,C++, COBOL, JAVA™, PHP, Perl, HTML, CSS, JavaScript, VBScript, ASPX,Microsoft .NET™ languages such as C#, and/or the like. Computing logicmay be compiled into executable programs or written in interpretedprogramming languages. Generally, functionality described herein can beimplemented as logic modules that can be duplicated to provide greaterprocessing capability, merged with other modules, or divided intosub-modules. The computing logic can be stored in any type ofcomputer-readable medium (e.g., a non-transitory medium such as astorage medium) or computer storage device and be stored on and executedby one or more general-purpose or special-purpose processors, thuscreating a special-purpose computing device configured to providefunctionality described herein.

Extensions and Alternatives

The methods and systems could be used for a wide variety of purposes,including admissions processes, market research, job interviews,academic testing (e.g., for oral examinations), tutoring and teaching(e.g., to allow instructors to answer questions posed by studentsoutside of class or office hours), dating sites (e.g., for interviewingdating prospects), or any other context where questions and answers canbe submitted asynchronously. The methods and systems disclosed herewithmay be implemented in connection with or as part of or using the methodsand systems disclosed in U.S. patent application Ser. No. 13/180,330,filed Jul. 11, 2011, and U.S. patent application Ser. No. 13/494,702,filed Jun. 12, 2012, which are incorporated herein by reference.

In any of the examples described herein, data can be transmittedsecurely. For example, an interviewee can be invited (e.g., by auniversity or prospective employer) to provide a secure username andpassword for authentication (e.g, via a web browser using secure HTTP(or HTTPS) and TLS (Transport Layer Security) or SSL (Secure SocketsLayer) protocols) in order to participate in a video interview and cansend encrypted data streams (e.g., via an encrypted Real-Time MessagingProtocol (RTMP) such as RTMPS or RTMPE) containing interview responses.

Systems, system components, and processes described herein may beprovided and used by a single institution or by multiple institutions.For example, a video interview system may provide college applicationinterview services for multiple colleges. A video interview system mayprovide customized user interfaces particular to each institution. Datafor multiple institutions may reside in a single data store or inmultiple data stores.

Many alternatives to the described video interview systems are possible.For example, although only a single client device and administratordevice are shown in FIGS. 1-3 for ease of illustration, the describedsystems can comprise multiple client devices and administrator devices,which can interact with the system one at a time or simultaneously.Multiple reviewers at multiple administrator devices can review andcomment on responses to expedite the review process. As another example,although interview servers are described that handle information andprovide services, alternative arrangements are possible in which clientdevices operated by interviewees communicate directly with devicesoperated by reviewers or other interview administrators. In sucharrangements, functionality such as storing and organizing interviewdata associated with various interviewees that is described as beinghandled by an interview server can be shifted instead to devicesoperated by reviewers or other interview administrators.

Although some examples refer to interviewers as being users of anadministrator device, actual interview questions can be delivered byother entities (e.g., according to instructions or scripts provided byan institution that wishes to interview an applicant). For example, anorganization that wishes to interview a native speaker of Chinese maywish to hire another native speaker of Chinese from outside theorganization to record interview questions, which can then be providedto the organization for subsequent use in video interviews. Furthermore,although reviewers and interviewers can be different users, in practicethe same user can act as both an interviewer and as a reviewer.

Interviewees can receive feedback. For example, interviewees can receivetext, audio, or video critiques in a manner similar to the way interviewquestions are received. Such features can be useful, for example, inpractice interviews coordinated by a career services department of auniversity.

While illustrative embodiments have been illustrated and described, itwill be appreciated that various changes can be made therein withoutdeparting from the spirit and scope of the claimed subject matter.

The embodiments of the invention in which an exclusive property orprivilege is claimed are defined as follows:
 1. A computer-implementedmethod for conducting a video interview by one or more servers incommunication with a client computing device, the method comprising:receiving a plurality of pre-recorded video prompts; generating aninterview script according to a prompt-ordering algorithm, wherein theinterview script comprises at least two of the plurality of pre-recordedvideo prompts; upon receiving an initiation request for an asynchronousvideo interview from the client computing device, transmitting a videoprompt from the interview script to be displayed at the client computingdevice as part of the asynchronous video interview; and receiving astreamed video response to the video prompt from the client computingdevice as part of the asynchronous video interview.
 2. The method ofclaim 1, further comprising causing the video response to be displayedin a reviewer user interface.
 3. The method of claim 1, furthercomprising receiving feedback from a reviewer.
 4. The method of claim 1,further comprising performing algorithmic analysis on the content of thevideo response, wherein the algorithmic analysis facilitates determininga response score for the video response.
 5. The method of claim 4,further comprising receiving a component score from the reviewer,wherein the response score is based at least in part on the componentscore.
 6. The method of claim 4, wherein the algorithmic analysiscomprises applying an audio analysis algorithm to audio data in thevideo response.
 7. The method of claim 6, wherein the audio analysisalgorithm comprises a speech recognition algorithm, the method furthercomprising generating a text transcript for the video response.
 8. Themethod of claim 7, further comprising analyzing the text transcript,wherein the response score is based at least in part on analysis of thetext transcript.
 9. The method of claim 6, further comprising analyzingaudio waveform peaks.
 10. The method of claim 6, further comprisinganalyzing inflection range.
 11. The method of claim 6, furthercomprising analyzing periods of silence.
 12. The method of claim 4,wherein the algorithmic analysis comprises applying a video analysisalgorithm to video data in the video response.
 13. The method of claim12, wherein the video analysis algorithm comprises a computer visionalgorithm.
 14. The method of claim 13, wherein the computer visionalgorithm is configured to detect one or more behaviors selected fromthe group consisting of: eye contact, facial expressions, and bodylanguage.
 15. The method of claim 4, wherein the algorithmic analysiscomprises determining a time duration of the video response.
 16. Themethod of claim 4, further comprising presenting results of thealgorithmic analysis in a reviewer user interface.
 17. Acomputer-implemented method for conducting a video interview, the methodcomprising: by a server, obtaining response preference data configuredto indicate a timing parameter for a video response to a video prompt inan asynchronous video interview; by the server, transmitting the videoprompt to be displayed in an interviewee user interface presented at aclient computing device; and by the server, receiving the video responsefrom the client computing device, wherein the video response is recordedin accordance with the response preference data.
 18. The method of claim17, wherein the timing parameter comprises a time limit for the videoresponse.
 19. The method of claim 17, wherein the timing parameterindicates that recording of the video response immediately followscompletion of playback of the video prompt at the client computingdevice, without a predetermined time delay.
 20. The method of claim 17,wherein the timing parameter comprises a predetermined time delay thatprecedes recording of the video response.
 21. The method of claim 17,wherein the response preference data is further configured to indicatewhether re-recording of the video response is permitted.
 22. Acomputer-implemented method for conducting a video interview, the methodcomprising: transmitting a video prompt to be displayed as part of anasynchronous video interview in an interviewee user interface presentedat a client computing device; transmitting an information supplement aspart of the asynchronous video interview to be presented at the clientcomputing device, wherein the information supplement relates to contentof the video prompt; and receiving a video response to the video promptand the information supplement from the client computing device.
 23. Themethod of claim 22, wherein the information supplement comprises aninformation form, the method further comprising receiving datacorresponding to user input in the information form.
 24. The method ofclaim 22, wherein the information supplement comprises a news item. 25.The method of claim 22, wherein the information supplement comprises adocument.
 26. A computer-implemented method for conducting a videointerview, the method comprising: by a server, performing analysis of avideo response recorded at a client computing device in an asynchronousvideo interview; and by the server, based on the analysis, automaticallyselecting a video prompt to be displayed at the client computing deviceas part of the asynchronous video interview.
 27. The method of claim 26,wherein the analysis comprises: determining a time duration of the videoresponse; and comparing the time duration of the video response with athreshold value.
 28. The method of claim 26, wherein the analysiscomprises analysis of keywords or a text transcript of the videoresponse.
 29. A computer-implemented method for conducting a videointerview, the method comprising: by a server, receiving informationabout the interviewee; by the server, performing analysis of thereceived information about the interviewee; and by the server, based onthe analysis, automatically selecting a video prompt to be displayed ata client computing device as part of an asynchronous video interview.30. The method of claim 29, wherein the received information about theinterviewee comprises personal information about the interviewee. 31.The method of claim 29, wherein the received information about theinterviewee comprises demographic information about the interviewee. 32.The method of claim 29, wherein the received information about theinterviewee comprises a language preference, and wherein theautomatically selected video prompt comprises content presented in alanguage that corresponds to the language preference.
 33. A servercomputer in communication with a client computing device, the servercomprising a processor and computer-readable storage media having storedthereon computer-executable instructions configured to cause the servercomputer to: transmit response preference data configured to indicate atiming parameter for a video response to a video prompt in anasynchronous video interview; transmit the video prompt to be displayedin an interviewee user interface presented at the client computingdevice; receive the video response from the client computing device,wherein the video response is recorded in accordance with the responsepreference data; and perform algorithmic analysis on the content of thevideo response, wherein the algorithmic analysis facilitates determininga response score for the video response.
 34. The server computer ofclaim 33, wherein the timing parameter indicates a preferred timeduration, and wherein the algorithmic analysis comprises comparing anactual time duration of the video response with the preferred timeduration.
 35. A computer-readable storage medium having stored thereoncomputer-executable instructions configured to cause a client computingdevice to: determine a timing parameter for a video response to a videoprompt in an asynchronous video interview; and display a user interfaceconfigured to: play the video prompt at the client computing device aspart of the asynchronous video interview; and display a set of one ormore user interface elements based at least in part on the timingparameter.
 36. The computer-readable storage medium of claim 35, whereinthe timing parameter comprises a time limit for the video response. 37.The computer-readable storage medium of claim 35, wherein the timingparameter comprises a predetermined time delay that precedes recordingof the video response.
 38. The computer-readable storage medium of claim35, wherein the timing parameter indicates that recording of the videoresponse immediately follows playback of the video prompt at the clientcomputing device without a predetermined time delay.
 39. Thecomputer-readable storage medium of claim 35, wherein thecomputer-executable instructions are further configured to cause aclient computing device to record the video response in accordance withthe timing parameter.
 40. A server computer in communication with aclient computing device, the server comprising a processor andcomputer-readable storage media having stored thereoncomputer-executable instructions configured to cause the server computerto: receive a plurality of pre-recorded video prompts; generate aninterview script according to a prompt-ordering algorithm, wherein theinterview script comprises at least two of the plurality of pre-recordedvideo prompts; upon receiving an initiation request for an asynchronousvideo interview from the client computing device, transmit a videoprompt from the interview script to be displayed at the client computingdevice as part of the asynchronous video interview; receive a streamedvideo response to the video prompt from the client computing device aspart of the asynchronous video interview; and perform algorithmicanalysis on the content of the video response, wherein the algorithmicanalysis facilitates determining a response score for the videoresponse.